Approximate balancing weights for clustered observational study designs.

In a clustered observational study, a treatment is assigned to groups and all units within the group are exposed to the treatment. We develop a new method for statistical adjustment in clustered observational studies using approximate balancing weights, a generalization of inverse propensity score weights that solve a convex optimization problem to find a set of weights that directly minimize a measure of covariate imbalance, subject to an additional penalty on the variance of the weights. We tailor the approximate balancing weights optimization problem to the clustered observational study setting by deriving an upper bound on the mean square error and finding weights that minimize this upper bound, linking the level of covariate balance to a bound on the bias. We implement the procedure by specializing the bound to a random cluster-level effects model, leading to a variance penalty that incorporates the signal-to-noise ratio and penalizes the weight on individuals and the total weight on groups differently according to the the intra-class correlation.

Using Balancing Weights to Target the Treatment Effect on the Treated when Overlap is Poor.

Inverse probability weights are commonly used in epidemiology to estimate causal effects in observational studies. Researchers often focus on either the average treatment effect or the average treatment effect on the treated with inverse probability weighting estimators. However, poor overlap in the baseline covariates between the treated and control groups can produce extreme weights that can result in biased treatment effect estimates. One alternative to inverse probability weights are overlap weights, which target the population with the most overlap on observed covariates. Although estimates based on overlap weights produce less bias in such contexts, the causal estimand can be difficult to interpret. An alternative to model-based inverse probability weights are balancing weights, which directly target imbalances during the estimation process, rather than model fit. Here, we explore whether balancing weights allow analysts to target the average treatment effect on the treated in cases where inverse probability weights lead to biased estimates due to poor overlap. We conduct three simulation studies and an empirical application. We find that balancing weights often allow the analyst to still target the average treatment effect on the treated even when overlap is poor. We show that although overlap weights remain a key tool, more familiar estimands can sometimes be targeted by using balancing weights.

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

A Trial Emulation Approach for Policy Evaluations with Group-level Longitudinal Data.

To limit the spread of the novel coronavirus, governments across the world implemented extraordinary physical distancing policies, such as stay-at-home orders. Numerous studies aim to estimate the effects of these policies. Many statistical and econometric methods, such as difference-in-differences, leverage repeated measurements, and variation in timing to estimate policy effects, including in the COVID-19 context. Although these methods are less common in epidemiology, epidemiologic researchers are well accustomed to handling similar complexities in studies of individual-level interventions. Target trial emulation emphasizes the need to carefully design a nonexperimental study in terms of inclusion and exclusion criteria, covariates, exposure definition, and outcome measurement-and the timing of those variables. We argue that policy evaluations using group-level longitudinal ("panel") data need to take a similar careful approach to study design that we refer to as policy trial emulation. This approach is especially important when intervention timing varies across jurisdictions; the main idea is to construct target trials separately for each treatment cohort (states that implement the policy at the same time) and then aggregate. We present a stylized analysis of the impact of state-level stay-at-home orders on total coronavirus cases. We argue that estimates from panel methods-with the right data and careful modeling and diagnostics-can help add to our understanding of many policies, though doing so is often challenging.

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. This study systematically reviewed the strength of evidence in the published COVID-19 policy impact evaluation literature. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26, 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation, assessing what impact evaluation method was used, graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigor to be actionable by policymakers. This was largely driven by the circumstances under which policies were passed making it difficult to attribute changes in COVID-19 outcomes to particular policies. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

A trial emulation approach for policy evaluations with group-level longitudinal data.

To limit the spread of the novel coronavirus, governments across the world implemented extraordinary physical distancing policies, such as stay-at-home orders, and numerous studies aim to estimate their effects. Many statistical and econometric methods, such as difference-in-differences, leverage repeated measurements and variation in timing to estimate policy effects, including in the COVID-19 context. While these methods are less common in epidemiology, epidemiologic researchers are well accustomed to handling similar complexities in studies of individual-level interventions. "Target trial emulation" emphasizes the need to carefully design a non-experimental study in terms of inclusion and exclusion criteria, covariates, exposure definition, and outcome measurement -- and the timing of those variables. We argue that policy evaluations using group-level longitudinal ("panel") data need to take a similar careful approach to study design, which we refer to as "policy trial emulation." This is especially important when intervention timing varies across jurisdictions; the main idea is to construct target trials separately for each "treatment cohort" (states that implement the policy at the same time) and then aggregate. We present a stylized analysis of the impact of state-level stay-at-home orders on total coronavirus cases. We argue that estimates from panel methods -- with the right data and careful modeling and diagnostics -- can help add to our understanding of many policies, though doing so is often challenging.

Layoffs and the mental health and safety of remaining workers: a difference-in-differences analysis of the US aluminium industry.

BACKGROUND: Relatively few studies have examined the effects of layoffs on remaining workers, although the effects of layoffs and downsizing events may extend beyond those employees who lose their jobs. METHODS: We examined the effects of layoffs on mental healthcare utilisation and injury risk among workers at 30 US plants between 2003 and 2013. We defined layoffs as reductions in the hourly workforce of 20% or more at each plant. Using a difference-in-differences approach, we compared the change in outcomes during layoffs versus the same 3-month period 1 year previously, accounting for secular trends with control plants. RESULTS: Our study population included 15 502 workers and 7 layoff events between 2003 and 2013. Layoffs were associated with only minor decreases in injuries (-0.006, 95% CI -0.013 to 0.001). The probability of outpatient visits related to mental health increased by 1% during layoffs (0.010, 95% CI 0.003 to 0.017), and the probability of mental health-related prescriptions increased by 1.4% (0.014, 95% CI -0.0006 to 0.027). Among women, the increase in outpatient visits was more pronounced (0.017, 95% CI 0.003 to 0.031). Increased prescription utilisation appeared attributable primarily to opioid use (0.016, 95% CI 0.005 to 0.027). CONCLUSION: Our results indicate an association between layoffs and remaining workers' mental health and safety, although changes mental healthcare utilisation may reflect both changes in underlying mental health and changes in care-seeking. Future research on concordance of service utilisation and underlying health may yield valuable insight into the experiences employed workers in the wake of layoffs.

Death and injury from motor vehicle crashes in Colombia

We report data on the distribution and determinants of road deaths and injuries for all victims in Colombia, with the aim of defining targets and priorities for highway death prevention in that country and other rapidly urbanizing nations. Using information from Colombia's Fund for the Prevention of Road Injury and the national death registry, we studied data on deaths and injuries from 1991 to 1995 for the nation as a whole and for the country's two largest cities, Santa Fe de Bogotá and Medellín. Deaths and injuries are rising in the nation as a whole. Of the deaths, 75 percent occur in urban areas, and 80 percent are in males. Pedestrians aged 15-34 are a peak subgroup. Thirty-four percent of deaths are attibutable to speeding and/or alcohol consumption. Death tolls are highest at night and on weekends. Specific priority targets for intervention are indicated by the fact that 75 percent of road deaths in Colombia occur in urban areas and that 80 percent of all victims are males

Death and injury from motor vehicle crashes in Colombia

We report data on the distribution and determinants of road deaths and injuries for all victims in Colombia, with the aim of defining targets and priorities for highway death prevention in that country and other rapidly urbanizing nations. Using information from Colombia's Fund for the Prevention of Road Injury and the national death registry, we studied data on deaths and injuries from 1991 to 1995 for the nation as a whole and for the country's two largest cities, Santa Fe de Bogotá and Medellín. Deaths and injuries are rising in the nation as a whole. Of the deaths, 75 percent occur in urban areas, and 80 percent are in males. Pedestrians aged 15-34 are a peak subgroup. Thirty-four percent of deaths are attibutable to speeding and/or alcohol consumption. Death tolls are highest at night and on weekends. Specific priority targets for intervention are indicated by the fact that 75 percent of road deaths in Colombia occur in urban areas and that 80 percent of all victims are males